Computational Simulations of Mediated Face-to-Face Multimodal Communication

Computational Simulations of Mediated Face-to-Face Multimodal Communication Melanie A. Baljko Doctor of Philosophy, November 2004 Graduate Department of Computer Science, University of Toronto Individuals who have little or no functional speech due to underlying physical disorder may instead use a computational device, called a Voice Output Communication Aid (VOCA), to produce synthesized speech. We describe a previously-unidentified set of tradeoffs that face the designers of Augmentative and Alternative Communication (AAC) systems, of which VOCAs are one possible component. On the one hand, the mode of synthesized speech that is afforded by a VOCA can be used to produce communicative actions that are more likely to be successfully interpreted than those produced using other modes, especially by unfamiliar communication partners — a benefit that can justify the often-sizeable effort that must be expended by individuals in order to use their VOCAs. On the other hand, the use of this so-called aidedmode can conflict with the simultaneous use of the other, unaided modes, such as facial expression, eye gaze, vocalization, and gesture — a negative effect on the interlocutor’s ability to producemultimodal communicative actions. These actions can be equally or evenmore effective than unimodal ones produced using synthesized speech alone, while also requiring less effort. The use of multimodal interfaces for VOCAs was first proposed by Shein et al. [1990]; prototypes have since been developed by Treviranus et al. [1991], Smith et al. [1996], and Keates and Robinson [1998]. We describe and formalize a previously-unidentified mechanism whereby a repertoire of modes of articulation affords a repertoire of mode strategies. The mechanism developed here accounts for the effects of conflict among the modes in a repertoire — a situation in which two or more modes rely on common underlying communicative effectors, as is the case with synthesized speech and gesture. We instantiated the mechanism computationally and used it to simulate the consequences of unimodal and multimodal VOCA interfaces on a simulated communicator’s repertoire of mode strategies. We show that, for the unimodal interfaces, empirical and anecdotal evidence agree with the simulation results. We also show, through the simulations, that the mechanism of mode conflict can have serious consequences for the utility of multimodal VOCA interfaces and thus the bottleneck-reduction hypothesis.